feat(server): implement log rotation based on size and retention #2452

Svecco · 2025-12-07T06:09:36Z

Redirected from issue #46 , the progress:

implemented log rotation based on size and retention as the title;
implemented configurable attributes and imported breaking changes;
added units and integration test in logger.rs and integration mod;
added documentations and imported new dependencies, etc.

Before merge:

Verified to comply with Apache specifications;
Add a relevant unit test for logging;
Confirm that the code has been optimized well?
Squash commits to a proper number after polishing;
Solve conflicts and confirm it is ready for being merged;

Plan:
Around the next several weekends, before Mid February.

hubcio · 2025-12-07T09:53:08Z

I think changes in core/message_bus are not related to log retention, looks like merge problem.

…retention - Configurable maximum log size and retention period from server configuration - Optimized single directory traversal for collecting file information - Eliminated redundant metadata() calls for improved performance - Enhanced error handling with detailed logging for all failure cases - Merged directory traversals into a single operation to improve performance Fixes apache#46, detail changes can be found at apache#2452.

Svecco · 2025-12-13T03:09:20Z

I think changes in core/message_bus are not related to log retention, looks like merge problem.

Yes, that was the case. I have just rebased it to the correct state.

core/server/src/log/logger.rs

…retention - Configurable maximum log size and retention period from server configuration - Optimized single directory traversal for collecting file information - Eliminated redundant metadata() calls for improved performance - Enhanced error handling with detailed logging for all failure cases - Merged directory traversals into a single operation to improve performance Fixes apache#46, detail changes can be found at apache#2452.

Svecco · 2025-12-27T12:31:00Z

@hubcio @spetz I've just squashed commits into one.

core/server/src/configs/defaults.rs

core/server/src/configs/system.rs

core/configs/server.toml

core/server/src/configs/system.rs

core/server/src/log/logger.rs

core/server/Cargo.toml

core/server/src/log/logger.rs

…retention - Configurable maximum log size and retention period from server configuration - Optimized single directory traversal for collecting file information - Eliminated redundant metadata() calls for improved performance - Enhanced error handling with detailed logging for all failure cases - Merged directory traversals into a single operation to improve performance Fixes apache#46, detail changes can be found at apache#2452.

hubcio

I tested your code manually. In server.toml i changed settings to:

max_file_size = "1 MB"
max_total_size = "10 MB"
rotation_check_interval = "1 s"

then I rebulilt server and started it via:
RUST_LOG=trace target/debug/iggy-server --with-default-root-credentials --fresh

and in other shell i run benchmark to generate logs:
target/debug/iggy-bench -T 1000MB pp -p 1 tcp

result:
I can never see more than 2 files, it looks like retention is ignored and 10 MB of max total size is not reached.

can you write integration test that would test for this scenario? see how TestServer is used, in integration module (e.g. core/integration/tests/server/specific.rs)

core/server/src/log/logger.rs

- implemented log rotation based on size and retention as the title; - implemented configurable attributions and imported breaking changes; - added units and integration test in logger.rs and intergration mod; - added documentations and imported new dependencies, etc. Details can be looked up at Apache Iggy apache#2452

Svecco · 2026-01-11T05:19:39Z

Hi, I've completed some initial modifications and added integration tests.

I first manually observed the runtime behavior locally by running in the terminal:
RUST_LOG=trace cargo run --bin iggy-server -- --fresh --with-default-root-credentials &&
cargo run --bin iggy-bench -- -T 1000MB pp -p 1 quic
I verified that the current log rotation behavior is working as expected by monitoring with
watch -n 0.1 "eza -la ~/Apache/iggy/local_data/logs"

eza -la with 10:100

.rw-r--r--@ 2.4M svecco 11 Jan 13:10 iggy-server.log
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.1
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.2
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.3
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.4
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.5
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.6
.rw-r--r--@  10M svecco 11 Jan 13:03 iggy-server.log.7
.rw-r--r--@  10M svecco 11 Jan 13:03 iggy-server.log.8
.rw-r--r--@  10M svecco 11 Jan 13:03 iggy-server.log.9

Additionally, the result of the simple integration test cargo test -p integration server::specific::log_rotation_should_launch -- --nocapture execution seems fine, by generating sufficient logs by sending a large volume of messages to trigger the log rotation mechanism.

Console

╭─ $ [svecco] ~/A/iggy git!(46-size-logs)
╰─ > cargo test -p integration server::specific::log_rotation_should_launch -- --nocapture
   Compiling libz-sys v1.1.23
   Compiling liblzma-sys v0.4.4
   Compiling hwlocality-sys v0.6.4
   Compiling libgit2-sys v0.18.3+1.9.2
   Compiling liblzma v0.4.5
   Compiling hwlocality v1.0.0-alpha.11
   Compiling compression-codecs v0.4.35
   Compiling async-compression v0.4.36
   Compiling async_zip v0.0.18
   Compiling git2 v0.20.3
   Compiling vergen-git2 v1.0.7
   Compiling server v0.6.1-edge.3 (/home/svecco/host/apache/iggy/core/server)
   Compiling integration v0.0.1 (/home/svecco/host/apache/iggy/core/integration)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7.71s
     Running unittests src/lib.rs (target/debug/deps/integration-375a7f260e09ad62)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/mod.rs (target/debug/deps/mod-0f48c9a972f7dba1)

running 1 test
Log directory verified: local_data_136642763006894046097605999160554396417/logs
Succeeded Verified Log Rotation
test server::specific::log_rotation_should_launch ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 1190 filtered out; finished in 2.12s

core/integration/tests/server/scenarios/log_rotation_scenario.rs

core/server/src/configs/validators.rs

- implemented log rotation based on size and retention as the title; - implemented configurable attributions and imported breaking changes; - added units and integration test in logger.rs and intergration mod; - added documentations and imported new dependencies, etc. Details can be looked up at Apache Iggy apache#2452

Svecco · 2026-01-17T14:39:42Z

Current code testing status:

Jan 18 => Cargo Test Console Results && 1MB:3MB:1s:7d Configured [PASSED]

╭─ $ [svecco] ~/A/iggy git!(46-size-logs M2)
╰─ > cargo test -p integration server::specific::log_rotation_should_launch -- --nocapture
   Compiling git2 v0.20.3
   Compiling compression-codecs v0.4.35
   Compiling async-compression v0.4.36
   Compiling async_zip v0.0.18
   Compiling vergen-git2 v1.0.7
   Compiling server v0.6.1-edge.4 (/home/svecco/host/apache/iggy/core/server)
   Compiling integration v0.0.1 (/home/svecco/host/apache/iggy/core/integration)
    Finished `test` profile [optimized + debuginfo] target(s) in 21.96s
     Running unittests src/lib.rs (target/debug/deps/integration-da023a48286e4fe0)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/mod.rs (target/debug/deps/mod-6bd13926aa022770)

running 1 test

     Size <-> Path && server::specific::log_rotation_should_launch::nocapture_observer
  1000 KB <-> local_data_205105475777995996131675610732819356417/logs/iggy-server.log.3
  1000 KB <-> local_data_205105475777995996131675610732819356417/logs/iggy-server.log.2
  1000 KB <-> local_data_205105475777995996131675610732819356417/logs/iggy-server.log.1
   365 KB <-> local_data_205105475777995996131675610732819356417/logs/iggy-server.log


     Size <-> Path && server::specific::log_rotation_should_launch::nocapture_observer
  1000 KB <-> local_data_205105475777995996131675610732819356417/logs/iggy-server.log.2
  1000 KB <-> local_data_205105475777995996131675610732819356417/logs/iggy-server.log.1
   365 KB <-> local_data_205105475777995996131675610732819356417/logs/iggy-server.log

test server::specific::log_rotation_should_launch ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 1195 filtered out; finished in 2.93s

╭─ $ [svecco] ~/A/iggy git!(46-size-logs A2)
╰─ > t
Sun Jan 18 11:58:53 AM HKT 2026

Jan 18 => Server && Bench Side Commands => Protocol = QUIC

╭─ $ [svecco] ~/A/iggy git!(46-size-logs)
╰─ > RUST_LOG=trace cargo run --bin iggy-server -- --fresh --with-default-root-credentials

╭─ $ [svecco] ~/A/iggy git!(46-size-logs)
╰─ > cargo run --bin iggy-bench -- -T 10000MB pp -p 5 quic

╭─ $ [svecco] ~/A/iggy git!(46-size-logs)
╰─ > set -x IGGY_CONFIG_PATH /home/svecco/Apache/.env/server.toml;

Jan 18 => Manual Watch && 10MB:100MB:1s:7d Configured: Expected Rotation

	Every 0.1s: eza -la ~/Apache/iggy/local_data/logs
	orion: Sun Jan 18 11:57:34 2026

.rw-r--r--@ 6.0M svecco 18 Jan 11:57 iggy-server.log
.rw-r--r--@  10M svecco 18 Jan 11:57 iggy-server.log.1
.rw-r--r--@  10M svecco 18 Jan 11:57 iggy-server.log.2
.rw-r--r--@  10M svecco 18 Jan 11:57 iggy-server.log.3
.rw-r--r--@  10M svecco 18 Jan 11:57 iggy-server.log.4
.rw-r--r--@  10M svecco 18 Jan 11:57 iggy-server.log.5
.rw-r--r--@  10M svecco 18 Jan 11:57 iggy-server.log.6
.rw-r--r--@  10M svecco 18 Jan 11:57 iggy-server.log.7
.rw-r--r--@  10M svecco 18 Jan 11:57 iggy-server.log.8
.rw-r--r--@  10M svecco 18 Jan 11:57 iggy-server.log.9

Jan 18 => Manual Watch && 10MB:0MB:1s:7d Configured: Expected Unlimited Archives

	Every 0.1s: eza -la ~/Apache/iggy/local_data/logs
	orion: Sun Jan 18 12:00:12 2026

.rw-r--r--@ 4.8M svecco 18 Jan 12:00 iggy-server.log
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.1
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.2
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.3
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.4
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.5
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.6
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.7
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.8
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.9
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.10
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.11
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.12
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.13
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.14
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.15
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.16
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.17
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.18
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.19
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.20
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.21
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.22
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.23
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.24
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.25
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.26
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.27
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.28
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.29
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.30
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.31
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.32
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.33
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.34
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.35
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.36
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.37
.rw-r--r--@  10M svecco 18 Jan 12:00 iggy-server.log.38

Jan 18 => Manual Watch && 0MB:500MB:1s:7d Configured: Expected Unlimited Size

	Every 0.1s: eza -la ~/Apache/iggy/local_data/logs
	orion: Sun Jan 18 12:02:09 2026

.rw-r--r--@ 695M svecco 18 Jan 12:02 iggy-server.log

Jan 18 => Manual Watch && 10MB:1MB:1s:7d Configured: Expected Failure

╭─ $ [svecco] ~/A/iggy git!(46-size-logs A2)
╰─ > RUST_LOG=trace cargo run --bin iggy-server -- --fresh --with-default-root-credentials
    Finished `dev` profile [optimized + debuginfo] target(s) in 0.18s
     Running `target/debug/iggy-server --fresh --with-default-root-credentials`
  ___                             ____                                      
 |_ _|   __ _    __ _   _   _    / ___|    ___   _ __  __   __   ___   _ __ 
  | |   / _` |  / _` | | | | |   \___ \   / _ \ | '__| \ \ / /  / _ \ | '__|
  | |  | (_| | | (_| | | |_| |    ___) | |  __/ | |     \ V /  |  __/ | |   
 |___|  \__, |  \__, |  \__, |   |____/   \___| |_|      \_/    \___| |_|   
        |___/   |___/   |___/                                               

Error: ConfigurationError(InvalidConfigurationValue)

…nition

Svecco force-pushed the 46-size-logs branch from d51fcec to 6b8546c Compare December 7, 2025 06:18

Svecco force-pushed the 46-size-logs branch from 6b8546c to 8390a77 Compare December 13, 2025 03:00

Svecco marked this pull request as ready for review December 13, 2025 03:04

spetz reviewed Dec 14, 2025

View reviewed changes

core/server/src/log/logger.rs Outdated Show resolved Hide resolved

spetz reviewed Dec 14, 2025

View reviewed changes

core/server/src/log/logger.rs Outdated Show resolved Hide resolved

spetz reviewed Dec 14, 2025

View reviewed changes

core/server/src/log/logger.rs Outdated Show resolved Hide resolved

spetz reviewed Dec 14, 2025

View reviewed changes

core/server/src/log/logger.rs Outdated Show resolved Hide resolved

spetz reviewed Dec 14, 2025

View reviewed changes

core/server/src/log/logger.rs Show resolved Hide resolved

spetz reviewed Dec 14, 2025

View reviewed changes

core/server/src/log/logger.rs Outdated Show resolved Hide resolved

spetz reviewed Dec 14, 2025

View reviewed changes

core/server/src/log/logger.rs Outdated Show resolved Hide resolved

hubcio reviewed Dec 15, 2025

View reviewed changes

core/server/src/log/logger.rs Outdated Show resolved Hide resolved

core/server/src/log/logger.rs Outdated Show resolved Hide resolved

core/server/src/log/logger.rs Outdated Show resolved Hide resolved

core/server/src/log/logger.rs Outdated Show resolved Hide resolved

Svecco force-pushed the 46-size-logs branch from b3a1d13 to 16fb847 Compare December 20, 2025 07:25

Svecco requested review from hubcio and spetz December 20, 2025 12:28

Svecco force-pushed the 46-size-logs branch 2 times, most recently from 8a52d32 to a1c8c0f Compare December 27, 2025 12:23

hubcio requested changes Dec 30, 2025

View reviewed changes

hubcio reviewed Dec 30, 2025

View reviewed changes

core/server/Cargo.toml Outdated Show resolved Hide resolved

hubcio reviewed Dec 30, 2025

View reviewed changes

Svecco force-pushed the 46-size-logs branch from a1c8c0f to d12c16b Compare January 1, 2026 09:24

Svecco requested a review from hubcio January 1, 2026 09:24

Svecco force-pushed the 46-size-logs branch 2 times, most recently from 9180baf to b2ca864 Compare January 1, 2026 12:58

hubcio reviewed Jan 2, 2026

View reviewed changes

core/server/src/log/logger.rs Outdated Show resolved Hide resolved

Svecco force-pushed the 46-size-logs branch from b2ca864 to 653de06 Compare January 11, 2026 03:55

Svecco requested a review from hubcio January 11, 2026 03:56

hubcio requested changes Jan 15, 2026

View reviewed changes

core/integration/tests/server/scenarios/log_rotation_scenario.rs Outdated Show resolved Hide resolved

core/server/src/configs/validators.rs Show resolved Hide resolved

chore(test): corrections && optimizations

c6a6f17

Svecco force-pushed the 46-size-logs branch from 948118a to c6a6f17 Compare January 17, 2026 14:41

Svecco added 2 commits January 18, 2026 10:06

chore(server): refined minor features && slightly optimized code style

b5f5020

chore(server): fixed where usize::MAX caused invalid archives

f682333

Svecco requested a review from hubcio January 18, 2026 04:03

chore(test): ensure zero value behavior aligns with IggyByteSize defi…

51647ad

…nition

feat(server): implement log rotation based on size and retention #2452

Are you sure you want to change the base?

feat(server): implement log rotation based on size and retention #2452

Conversation

Svecco commented Dec 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hubcio commented Dec 7, 2025

Uh oh!

Svecco commented Dec 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Svecco commented Dec 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hubcio left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Svecco commented Jan 11, 2026

Uh oh!

Uh oh!

Uh oh!

Svecco commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Svecco commented Dec 7, 2025 •

edited

Loading

hubcio left a comment •

edited

Loading

Svecco commented Jan 17, 2026 •

edited

Loading